{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* By: Jacques Joubert\n",
    "* Email: jacques@quantsportal.com\n",
    "* Reference: Advances in Financial Machine Learning, Marcos Lopez De Prado, pg 101\n",
    "\n",
    "\n",
    "# Chapter 6 Ensemble Methods\n",
    "\n",
    "## Question 1:\n",
    "**Why is bagging based on random sampling with replacement? Would bagging still reduce a forecast’s variance if sampling were without replacement?**\n",
    "\n",
    "Sampling without replacement is called Pasting and with replacement is Bagging. Pasting is designed for very large data sets that can afford to make use of sampling without replacement. Both have the same purpose in mind - to create a diverse set of models. These diverse models are then used in an ensemble method which has a similar bias but a lower variance than a single predictor trained on the original training set. One may also observe an increase in bias and a lowering in variance from an ensemble method.\n",
    "\n",
    "Bagging ends up with a slightly higher bias than pasting which results in the predictors being less correlated (for a small dataset), thus the ensemble's variance is reduced. Overall bagging is preferred as it often leads to better models, however for a large dataset, using Pasting could lead to less correlated predictors because there won’t be any duplicates in the observations.\n",
    "\n",
    "## Question 2:\n",
    "**Suppose that your training set is based on highly overlapping labels (i.e., with low uniqueness, as defined in Chapter 4).**\n",
    "\n",
    "**(a) Does this make bagging prone to overfitting, or just ineffective? Why?**\n",
    "\n",
    "“Redundant observations have two detrimental effects on bagging. First Samples drawn with replacement are more likely to be virtually identical, even if they do not share the same observations. This makes $\\overline{p} \\approx 1$, and bagging will not reduce variance, regardless of the number of estimators $N$.” (Advances in Financial Machine Learning, pg 97, 6.3.3 Observation Redundancy)\n",
    "\n",
    "The advantage of using Bagging lays in its ability to reduce forecast variance and thus prevents overfitting. The variance of the bagged prediction is a function of the number of bagged estimators, the average variance of a single estimator’s prediction, and the average correlation among their forecasts.\n",
    "\n",
    "Models that are trained on the same type of data are likely to make the same type of errors. When there are many overlapping samples (low uniqueness) then it results in models with poor diversity (high correlation).\n",
    "\n",
    "Bagging is only effective to the extent that the average correlation among forecasters is less than 1. One of the goals of sequential bootstrapping (Chapter 4) is to produce samples as independent as possible, thereby reducing the avg correlation, which should lower the variance of bagging classifiers.\n",
    "\n",
    "<div align=\"center\">\n",
    "  <img src=\"https://raw.githubusercontent.com/hudson-and-thames/research/master/Chapter6_EnsembleMethods/images/ch6_bagged_prediction.png\" width=\"500\"><br>\n",
    "</div>\n",
    "\n",
    "The plot above (Advances in Financial Machine Learning, pg 95) shows that models that are highly correlated (overbar p), fail to reduce the variance. It is better to have a few diverse models than a large number of correlated ones.\n",
    "\n",
    "Thus training models on data that has a low average uniqueness, makes the bagging ensemble more ineffective. It also leads to the out-of-bag accuracy being grossly over-inflated.\n",
    "\n",
    "**(b) Is out-of-bag accuracy generally reliable in financial applications? Why?**\n",
    "\n",
    "“The second detrimental effect from observation redundancy is that out-of-bag accuracy will be inflated. This happens because random sampling with replacement places in the training set samples that are very similar to those out-of-bag. In such a case, a proper stratified k-fold cross-validation without shuffling before partitioning will show a much lower testing-set accuracy than the one estimated out-of-bag. For this reason, it is advisable to set StratifiedKFold(n_splits=k, shuffle=False) when using that sklearn class, cross-validate the bagging classifier, and ignore the out-of-bag accuracy results. A low number k is preferred to a high one, as excessive partitioning would again place in the testing set samples too similar to those used in the training set.” (Advances in Financial Machine Learning, pg 97, 6.3.3 Observation Redundancy)\n",
    "\n",
    "\n",
    "## Question 3:\n",
    "\n",
    "**Build an ensemble of estimators, where the base estimator is a decision tree.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn.datasets import load_iris, make_classification\n",
    "from sklearn.ensemble import RandomForestClassifier, BaggingClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create data\n",
    "X, y = make_classification(n_samples=2000, n_features=30,\n",
    "                            n_informative=15, n_redundant=10,\n",
    "                            random_state=42, shuffle=True)\n",
    "\n",
    "# Split data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, shuffle=True, stratify=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out-of-bag score on training: 0.8464285714285714\n",
      "Test Score: 0.8533333333333334\n"
     ]
    }
   ],
   "source": [
    "# Create model\n",
    "bag_clf = BaggingClassifier(DecisionTreeClassifier(splitter=\"best\", max_leaf_nodes=None),\n",
    "                            n_estimators=500, max_samples=100, bootstrap=True, n_jobs=-1, oob_score=True)\n",
    "\n",
    "# Fit and score\n",
    "bag_clf.fit(X_train, y_train)\n",
    "y_pred = bag_clf.predict(X_test)\n",
    "y_pred = bag_clf.predict(X_test)\n",
    "\n",
    "print('Out-of-bag score on training: {}'.format(bag_clf.oob_score_))\n",
    "print('Test Score: {}'.format(accuracy_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(a) How is this ensemble different from an RF?**\n",
    "\n",
    "A random forest also makes use of a bagging method and thus it has the same hyperparameters as a decision tree and a bagging classifier. However the key difference is that it introduces extra randomness when building trees as it splits the data on the best feature from a random subset of features. This results in greater tree diversity, thus lowering the variance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b) Using sklearn, produce a bagging classifier that behaves like an RF. What parameters did you have to set up, and how?**\n",
    "\n",
    "First we build a random forest classifier and then we fit a bagging classifier which has its parameters adjusted to replicate the random forrest. In order to do this, we need to set the `splitter` hyperparameter in the decision trees to random. Consider using the `random_state` parameter to your classifiers to maybe get the exact same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Random Forest\n",
    "rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1, oob_score=True)\n",
    "rnd_clf.fit(X_train, y_train) \n",
    "y_pred_rf = rnd_clf.predict(X_test)\n",
    "\n",
    "# Bagging -> RF\n",
    "bag_clf = BaggingClassifier(DecisionTreeClassifier(splitter=\"random\", max_leaf_nodes=16),\n",
    "                            n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, oob_score=True)\n",
    "bag_clf.fit(X_train, y_train)\n",
    "y_pred_bagging = bag_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random Forest Classifier\n",
      "Out-of-bag score on training: 0.8657142857142858\n",
      "Test Score: 0.8633333333333333\n",
      "\n",
      "Bagging Classifier\n",
      "Out-of-bag score on training: 0.8535714285714285\n",
      "Test Score: 0.865\n"
     ]
    }
   ],
   "source": [
    "print('Random Forest Classifier')\n",
    "print('Out-of-bag score on training: {}'.format(rnd_clf.oob_score_))\n",
    "print('Test Score: {}'.format(accuracy_score(y_test, y_pred_rf)))\n",
    "print('')\n",
    "\n",
    "print('Bagging Classifier')\n",
    "print('Out-of-bag score on training: {}'.format(bag_clf.oob_score_))\n",
    "print('Test Score: {}'.format(accuracy_score(y_test, y_pred_bagging)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4\n",
    "**Consider the relation between an RF, the number of trees it is composed of, and the number of features utilized:**\n",
    "\n",
    "**(a) Could you envision a relation between the minimum number of trees needed in an RF and the number of features utilized?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following [post](https://stats.stackexchange.com/questions/36165/does-the-optimal-number-of-trees-in-a-random-forest-depend-on-the-number-of-pred/36183) on cross-validated has quite a good answer:\n",
    "\n",
    "\"*Random forest uses bagging (picking a sample of observations rather than all of them) and random subspace method (picking a sample of features rather than all of them, in other words - attribute bagging) to grow a tree. If the number of observations is large, but the number of trees is too small, then some observations will be predicted only once or even not at all. If the number of predictors is large but the number of trees is too small, then some features can (theoretically) be missed in all subspaces used. Both cases results in the decrease of random forest predictive power. But the last is a rather extreme case, since the selection of subspace is performed at each node.*\"\n",
    "\n",
    "I ran some empirical tests to validate this. First I created synthetic data using the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create data\n",
    "X, y = make_classification(n_samples=20000, n_features=50,\n",
    "                            n_informative=10, n_redundant=0,\n",
    "                            random_state=42, shuffle=True, n_classes=2, class_sep=1.0)\n",
    "\n",
    "# Split data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, shuffle=True, stratify=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 50 features, with only 10 of them being informative. There is a clear class separation and 20000 observations.\n",
    "\n",
    "Next, I fit a random forest which is limited by the number of trees, and the number of features it may use (n_estimators, max_features).\n",
    "\n",
    "**Warning:** This code takes long to run! Reduce the search space to have a faster run. It took 3 hours on my 8 core i7 8th gen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# max_trees = 100\n",
    "# max_feat_used = 50\n",
    "\n",
    "# store = []\n",
    "# for num_trees in range(2, max_trees, 2):\n",
    "#     print(num_trees)\n",
    "#     for num_feat in range(1, max_feat_used, 2):\n",
    "#         rnd_clf = RandomForestClassifier(criterion='entropy', n_estimators=num_trees, max_features=num_feat, n_jobs=-1)\n",
    "#         rnd_clf.fit(X_train, y_train) \n",
    "#         y_pred_rf = rnd_clf.predict(X_test)\n",
    "        \n",
    "#         store.append([num_trees, num_feat, accuracy_score(y_test, y_pred_rf)])\n",
    "\n",
    "# # Pivot and save results\n",
    "# results = pd.DataFrame(store, columns=['N', 'F', 'Score'])\n",
    "# pivot_results = results.pivot(index='N', columns='F', values='Score')\n",
    "# pivot_results = pivot_results.sort_index(ascending=False)\n",
    "# pivot_results.to_csv('relational_data_small2.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution:** Just import this data which I saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load saved results\n",
    "pivot_results = pd.read_csv('relational_data.csv', index_col=0)\n",
    "pivot_results = pivot_results.sort_index(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAG5CAYAAABY5iZxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XmcZGV59vHfNT07M+yCsggIJK4JyoCvRlFwQxEUAwY3BI2jcQnRaERFUYzghmiMBkcFlKgorkRwF1BckMEtoqhAkH3YGQZm7b7eP85pqOmtqnvqPFNdc33nU5+pOlXn3KeqTlXf9Sz3kW0iIiIiIjaWGRt7ByIiIiJi05aENCIiIiI2qiSkEREREbFRJSGNiIiIiI0qCWlEREREbFRJSCMiIiJio0pCGuuR9GRJ123A+qdKens392mMGJa0xzj3vUjSd5uM35QNfe27EP9QSddKWiHp0RtrPzYlki6SdNRGij1f0rmS7pL0hY2xD9ORpLdLOrW+vockt9z3XUkv6nK8f5d0Rje3GdGLkpD2IUlXS1pZJxY3STpD0oIG4hwl6aLWZbZfZfvd3Y7VKdufs/30bmxL0jvr5Pf5Lctm1st27UaMHvNB4LW2F9j+1fBCSQ+uj6XhiyXd03L7iRtxn9cj6TpJT57iusPv7a8lqWX5eyV9qms72Tv+Adga2Mb2C0beWSdCa0e892/Y0KAb8h5103if5dYEUNJTJV3der/td9t+1VjbtP10259rZo8j+lsS0v51sO0FwF7Ao4G3bOT9ma5uB94laWBj78hkSJo5hdV2AS4budD2NXWSuqA+pgD+tmXZj8eIP61erxF2Bg7f2DsxGZJmSJrs9/kuwB9tr5vgMZ9rfe9tf2gDdrMrpnhsR0SPS0La52zfBHyHKjEFQNIcSR+UdI2kZXU3+7yx1pd0rKQrJd0t6feSDq2XPww4FXhc3XJyZ738DEn/3rL+KyRdIel2SedI2qHlPkt6laQ/S7pT0seGW6bqrrAL6+7EWyV9ccSuPXWc9dZrta1j/LOkq+rtfGCSf7i/DawBXjzO63OBpH9suT1W/FfX+3q3pHdL2l3STyUtl/QlSbNHbPOt9b5e3dr9N9H7prq7X9KbJd0EnD7Gvs6QdJykv0i6WdJnJW1Rb3cFMAD8RtKVk3h9hrf93/X78G1J9wBPlDRX0odUDQNYJunjkua2rHOIpN/U7+FFkh454jW4oX6NLu+kRU1Vt/MOwLdaW/NUDUW4rI7zQ0l/3WZT72ecHyFjtZippcVPVevaWZK+UO/Db+r3+zhJt9Tv3VNHbHZPSUvrY/1rkrZq2fbfSfp5ve+/lrRfy30X1cfTz4B7gAePsb+PqD9Hd0r6X0kH1cvfA7wVeFG9ny9t85qM3O6Wkk6XdGP9/E8Y/lxJ2lPS+ao+87dKOlPSFvV9o96jDl/TL9av6d3Ai+tj+a2qvpturV/zrerHz5f0eUm31c/7F5K2nczzq7ezBfA/QGsPwXaaoAtdLUMw6mNuZM/CE+r7JnpfHyLpx6q+L74DbDPZfY+YjpKQ9jlJOwHPBK5oWfxe4K+oktQ9gB2Bd4yziSuBJwJbAO8C/lvSg2z/AXgV8LO65WTLMWIfAJwEPB94EPAX4KwRD3s2sA/wN/XjnlEvfzfwXWArYCfgox2uN5ZDgUXAY4DnAC+r9+/B9R+EUX/IWxh4O3C8pFkTPG4izwD2Bv4f8G/AEqoEd2fgkUBrd+kDgW2p3pOXAktaEqh279sDqbpgdwEWj7EfR9WX/YGHAAuA/7S9ekTL5+5TfJ4vpDpGFgI/Az4A7Eb1Hu0J7Aq8DUDSPsAngX+k+oN7GvANSbMlPQJ4JfAY25tTHb/XtAtedzvfADxzuDVP1Q+nM4HXAQ8Avg+c0+a9/BKwGnjJpJ79/Z4DfBrYkqrF+fvAOqrPwEnAf414/JH1ZQdAwCkAknYGzgGOp3pfjwW+Kqk1QXkJ1fG8ObDe+GNVP3S+CZxL9dxfD3xR0h6230aVeA+3gH5mks/xTGAlsDvVsX0QcPRwaODfqY7Hh1Mda2+Hsd+jDuMdCnye6nvoi/VzOQjYj+r7YQXwH/Vjjwbm18u3AV4NrJrk88P2XcDBQGsPwc2TWP8RLb0K/wb8nuoHX7v39YvAz6m+B05i6sdhxLSShLR/fb1uTbgWuJnqyw9JokpWXm/7dtt3AycCR4y1Edtn277B9pDtLwJ/BvbtcB9eBJxm+5e2V1MNG3ic1h+z9V7bd9q+Bjif+1ty11IlVjvYXmV7vbGqE6w3lvfVz/Ua4MPUCWDdFb1lvXxcts8BbqFKnqbi/baX274M+B3wXdtX1X/wvkU1pKLV2+sk8UKqZOL5Hb5vQ8Dx9borx9iPFwEfqmOvoHo/jlD3ukC/Zvtntoeo3r9XAP9i+w7by6n+uA7v72Lg47YvsT1o+7R6+T5Uydtc4BGSZtr+P9tXTXGfjgDOsf1D22upkvotgMdOsI6pEv2p/gi5wPb3667ws6mSjvfXt88C9tD6Y7o/Y/v3tu+p4x5Rv99H1vv+nfrz923gN8CBLeueZvsPtteO0fX+d8Bs4AP1/d+nOt7G/KyP44X1j7bhy3aSdgSeSnUs3mt7GdXn6ggA23+y/QPba+oE7hTgSZOIOZaLbP9P/TqspPox/Fbb19teRfVD6PC6lXYtVTK3R31sLa2P941C0pOovn+fU39ux31fJT0E+Fvu/xxfAJy3sfY9oqQkpP3rubYXAk8GHkr1BQ1VS8l84NLhPzJU3dIPGGsjko6su5SGH/vIlm21swNVqygA9R+F26ha9obd1HL9XqpWO6haFAT8ou76etmIbY+33liubbn+l3q/Jus4qta9ue0eOIZlLddXjnG7dd/vqBOTYcP728n7dkv9x3k8670f9fWZwPadPpE2Wl/nBwJzqFqEhvf3m8B29f27AG9uTXaoWhB3tP1H4F+BE4Cb667aB05xn0Yeg0NULYk7jrsG9/0IuZmp/QgZ+f7eUscdvg3rv+cjj8853N/S/YIRr9H/Y/3jt3XdkXagat1zy7K/0Oa5j/D5+kfb8OXmer/mAMta9utj1MeRpAeqGopyvaTlwBl0/p0xnpHP88HA/7TE/996+XZ1vO8Dw/vw3nF+dA3Vl5E/OmZRJbUbrP7xfRbwEtvDvVQTva87ALfZvrdlM62f2Yi+lYS0z9WtbGdQzaAGuJXqj+IjWv7IbNHSZXsfSbtQdau+lmom7pZULXzDM5A9cp0RbqD68h3e3mZUXWjXd7DfN9l+he0dqLpvP65xSj11YOeW6w+u92tSbH+PatjDq0fcdQ9VojhsqonTsK3q12nY8P528r5N6v2ot72O9ROoDdEafxnV2Nu/HrG/W9T3Xwu8a0SyM9/2lwBs/7ftv6Pq8h+gal2d7D7A6GNwBlVXbttjkOoHyHGs/yNkvfe7TnQ2dIzfyONzNdVkumuB00e8RpvZ/kDL4yd6z28Adq5bW1u338lzn8i1VD8Ct27Zr81t/019//vq5/AoV0MujuL+74yx9rmT13TkOtcBTxvx2sytvzfW2H6n7YcBT6Dq7h9Viqn+kXA91VCSVrtxfxLY7jM1rvpz/HWqFurWUnQTva83Atto/TH9Ew0piugbSUg3DR8Gnibpb+sv4U8Cp0jaDkDSjpLGGoO5GdUX8i31446maiEdtgzYSSMm5bT4AnC0pL0kzaHqYr7Y9tXtdljS4fX4V4A76v0YmmCVibxJ0lb12K1jqMZoTcXbqFpuW/0aeF49kWIP4OVT3Hard9VjKZ9INVb27Em+b+P5AvB6SbvVXcYnAl8co6t3g9keBD4FfFjSA1TZSdJwSa5PAq+RtE993wJJB0vaTNLDJO1fHzMr68sQ3DepaKL9XUY1ZnHYl4BDVE36mgW8CbgbuLiD5/B94E+sP4bvcmChpGfU2zue0S1sk3WkpIfWCcy7gC/VrZpnAodKepqkAVWTxPZXy8TANn5K9YPjXyXNUjWm+1lM/fgHwPa1wIXAByVtrmqC0R66f2LOQqok8676M/fGEZsY+R5N5TU9FThR9fjveijBIfX1AyQ9sv7xsZyqtXO8744vAm+vP0sz6uPzmcBXWvZ1W0kL2+zPWE4HfuPR42THfV9tXwn8Fnhn/R2wH9VY2Yi+l4R0E2D7FuCz3D8B5s1UrX0/r7vUvg+Mmnls+/fAyVQTVJYBjwJ+0vKQH1JN2rhJ0q1jrP99qskMX6H65b87nY9f2we4WNXs73OAYzZgHOE3gEupksdzqSactNbX7KgFwvZPgF+MWHwKVUvgMuAzwIbWILyJKgG/od7Wq2xfXt/X0fs2gdOo/hj+CPg/qoker9vA/Z3Iv1K1NP0CuItqktqeALZ/DvwT1QSfO6gSv+FKBnOoJtzcSvV6bEU9GYqqNbH1GBzpRKqE/k5J/+Jq3O5L6zi3UI2/PKQeT9qJt1F1n1Pv9x1Ur9lnqFrXbmf94SNTcSbw31SfkQHgX+pYV1O17r293vdrqF7Tjr63XY3bPphqktWtVJN+Xmj7zxu4v1C9V5tRTdS5g2qs7HDvwPFU48zvovrsfmXEuiPfo6m8ph+iGrLyA1Vj5X9K9Z0BVbf3V6mS0eFJZZ8fZzvHA5fU699BPc7Z1aRNbP+u3v+r6/3dbpztrKdu5T2calxr60z7x3Xwvh5BNf73dqrj78yW7Q4Mb6eT/YiYTrT+8KKI/qLqLCp7tozfimlMVbmdM23/YGPvS0REdE8S0uhrSUgjIiJ6X7rsIyIiImKjSgtpRERERGxUaSGNiIiIiI2qW2do6boVb3xOkabbgX33LhGmst1O7R/TJdqyo8mg3Yk1Y0Or3kzCvKlUX5mCGeV+q2nmeFWzGjBj1OnZmzNQ8Otl7eoiYVTq+AMY7Ho1rmhKwe+LotSfz2vWdnuq/aOatfbWq7qW48za9iEb/fl0Q38ebRERERExbfRsC2lEREREXxoa3Nh70HPSQhoRERERG1UjCWl9CraXSTpX0m8k/VLSWZKe3Ga9xZKWSlp62m+vbmLXIiIiIjYuD3Xv0iea6rL/NNUpA08CDqM6hduPgeMkPcr2R8dayfYSYAmUm9QUERERUdRQ/ySS3dJUQrq37aPr6xdJ+rntd0j6EdX5xMdMSCMiIiJi09NUQrpW0u62r5T0GGANgO3V9akcIyIiIjZJ7qOu9m5pKiF9E3C+pDXAAPACAEkPAL7ZyQbW3Xh3Q7u2voHBcjPdNFCwXuealeVizS34vNYVqjc5Z7MicQCYNadcrH5V6jUs+UekZB3XkvVpB9eWi9WPSv4dKTkTfFNL0NJlP0oj33i2fyhpF2Ab27cCSPqs7SOBf2siZkRERERMT40kpJLOabk+fPUASVsC2D6kibgRERERPW9TaxHuQFN9QjsDlwGfAgwI2Ac4uaF4EREREdNDCuOP0lRh/L2BS4G3AXfZvgBYaftC2xc2FDMiIiIipqGmxpAOAadIOrv+f1lTsSIiIiKmlXTZj9Jokmj7OuBwSQdRFcePiIiI2LRllv0oRVotbZ8LnFsiVkRERERMLz3bjT5jfqFdmzuvTBzABevvSfOLxWL23GKhvPqeInGK1iEdXFcuVsmB9AXrq0pNDYcfoWirRsFYJbsPS71X/apP67i65PdgD0hh/NF6NiGNiIiI6Evpsh8lP1UjIiIiYqNKC2lERERESemyH6WRFlJJX5X0YkkLJrneYklLJS09/fLrmti1iIiIiI1raLB7lz7RVJf9Y4HnAtdI+pKkQyXNbreS7SW2F9ledPRDd2po1yIiIiKilzSVkN5s+zBgV+B/gFcA10s6XdLTG4oZERER0fs81L1Ln2gqITWA7eW2z7T9LOChwMXAsQ3FjIiIiOh9Q0Pdu/SJpiY1rRi5wPZtwKn1pS2vK/Qir1pZJg7AujV9Gcur7i4WS3MXFolTsiaeZqTYxYYq9X5poD/ngfZjDchitWkB+vUzXDDZ0cy2o/qizzXyKbK938hlkj7bRKyIiIiIaSVd9qM08nNf0jkjFwH7S9oSwPYhTcSNiIiI6Hl91NXeLU31P+0MXAZ8imo8qYBFwMkNxYuIiIiIaaqphHRv4BjgbcCbbP9a0krbFzYULyIiImJasPunfmi3NJKQ2h4CTpF0dv3/sqZiRUREREwrfTT2s1saTRJtXwccLukgYHmTsSIiIiJieirSamn7XODcyawzY06ZBtV1F17MzP0fVyQWty2DLbcpE2tdwbJFczYrFquYwbVobsHnlQHuG0SzypWM8doyJdVKPifWFCxJV6h0lj1UtpRQqRavkt8Vs+aUi9VHp8DsSL7zR9nku9GLJaNQLhmNDZZkNMZSKhmNDdeXyWj0jxwzo2zyCWlEREREUZtai3AH+vT0EhERERExXTSSkEr6m5brsyQdJ+kcSSdKmt9EzIiIiIhpIWdqGqWpFtIzWq6/F9iDqij+PCY4l72kxZKWSlp6+u+vbWjXIiIiIjaioaHuXfpEU2NI1XL9KcA+ttdK+hHwm/FWsr0EWAJw96sOdEP7FhERERE9pKmEdAtJh1K1wM6xvRbAtiUl0YyIiIhNVx91tXdLUwnpj4BD6us/l7S97WWSHgjc2skGBu8uU15l8JwLmfPsJxaJxepVZeIAbP2gcrFKdhkMljkuPDCrSBwAZpSbW9iXNWOh2DGoQjU0SytaIqnU8e4hUJlYHixY97nkMTi4tlysQu9Vz+ijrvZuaerUoUeNXCbps7aPpOrC7xnFktGIiChnU0twIqa5RhJSSeeMsfgASVsC2D5kjPsjIiIi+l9aSEdpqu1/Z+Ay4FOAqSY57UM10z4iIiJik2WnMP5ITfVp7A1cCrwNuMv2BcBK2xfavrChmBERERExDTU1hnQIOEXS2fX/y5qKFRERETGtpMt+lEaTRNvXAYdLOghY3mSsiIiIiGkhZZ9GKdJqaftc4NwSsSIiIiJieunZbvSZ2y8oEmfwkl8xsM+ji8Ri7vwycQBmDBSMVbC8ysDcImE0q2BdxqJ1XAvWFaRcLVf3YYUfFTwsXLC1ptTzsvu0Nmi/KljLtSeky36UTf5TVCwZjYiIiIB02Y+hD9sVIiIiImI62eRbSCMiIiKKSpf9KE2dqWk+8FqqovgfBY4AngdcDpxge0UTcSMiIiJ6XrrsR2mqy/4MYHtgN6rZ9YuAD1Cdsem/xltJ0mJJSyUtPe23f2lo1yIiIiKilzTVZf9Xtp8vScCNwFNtW9JFwG/GW8n2EmAJwIo3HOKG9i0iIiJi40mX/ShNF8a3pPNsu+V2Es2IiIjYdCUhHaWphHSppAW2V9h+2fBCSbsDd3eygXXLygwzXffNHzP7iY8qEksP2qVIHADWrCwWyjPL1ezUnM2KxPGaVUXiAGh2mdqqAAyUqw1adIzU2kL1VUu+frPKzTlVyYIr/fiHuORxsXZ1uVgFlayFG72pkW8h2/84cuKSpM/avhJ4YhMxp6pUMhoREREBVD/Yu3XpgKQDJf1R0hWSjh3j/l0k/UDSbyVdIGmnlvteKunP9eWlXXwV1tPULPtzRi4C9pe0ZX37kCbiRkRERPS8gj0FkgaAjwFPA64DLpF0ju3ftzzsg8BnbX9G0gHAScBLJG0NHE81Od3ApfW6d3R7P5vqE9oZuAz4FNUTENWTObmheBEREREx2r7AFbavApB0FvAcoDUhfTjwhvr6+cDX6+vPAL5n+/Z63e8BBwJf6PZONjVwaG/gUuBtwF22LwBW2r7Q9oUNxYyIiIjofV3ssm8tmVlfFo+ItiNwbcvt6+plrX5DVS8e4FBgoaRtOly3KxppIXU1OvkUSWfX/y9rKlZERETEtNLFLvvWkpkb4I3Af0o6CvgRcD0wuIHbnJSmyz5dBxwu6SBgeZOxIiIiImKU66mGUg7bqV52H9s3ULeQSloA/L3tOyVdDzx5xLoXNLGTRWp92D7X9ltLxIqIiIjoaWVn2V8C7ClpN0mzqU7nvt7kc0nbShrOCd8CnFZf/w7wdElbSdoKeHq9rOt6txt9XZn6+Zo3r0gcAArWtmTh1uViFax5yrzNy8QZKPfR8No1xWKpWCSK1mbU3EL1aQfXFYkDlK3XWfB4L1XyVCr4nArWBi15DJaskaySn61eUPDzbXudpNdSJZIDwGm2L5N0ArDU9jlUraAn1Scv+hHwmnrd2yW9myqpBThheIJTt/VuQhoRERERG8z2ecB5I5a9o+X6l4Evj7PuadzfYtqYJKQRERERJfXjGcs2UBLSiIiIiJJcZljidFLwBMYVSRtamiAiIiIi+khTpw4db0aNgGdNsN5iYDHAKXv/FUftvkMDexcRERGxEaXLfpSmuuxvAf7C+pN6h08hut14K7UWd73zH/ZPe3ZERET0nySkozSVkF4FPMX2NSPvkHTtGI+PiIiIiE1UUwnph4GtgFEJKfD+TjYwY/NCNQxL/kopWWdNBYcHFx+JXEDB40KzZheLVfK40Mxyz6tkLddSTMHvps6Ka3dHoc9W0c9VQUWfV8njomQt3F5Q8rWdJpo6l/3HRi6T9FnbR9r+aBMxIyIiIqaFdNmP0tSkpnNGLgL2l7QlgO1DmogbEREREdNPU23kOwOXAZ/i/slMi4CTG4oXERERMT2kDukoTQ0o2xu4FHgbcJftC4CVti+0fWFDMSMiIiJ639BQ9y59oqkxpEPAKZLOrv9f1lSsiIiIiJjeGk0SbV8HHC7pIGB5k7EiIiIipoU+atnsliKtlrbPBc4tESsiIiKip6Xs0yg9242ugUL1Ekv+SilZZ63gwa65C4vFYtU9ZeLM3axMHIDZ88rFWru6WCiX/MKd0Y/FcMtRybrFswp9D5Z8TqX+XkH/JjL9+ryiYz2bkEZERET0Iw9llv1ISUgjIiIiSsoY0lHSzxURERERG1VTZ2qaARwF/D2wEzAI/Ak4ta5JGhEREbFpypjZUZpqIf008GDgJOB84Jv1suMkvW68lSQtlrRU0tLTf39tQ7sWERERsRENuXuXPtHUGNK9bR9dX79I0s9tv0PSj4BfAx8dayXbS4AlAHe/6sD+eZUjIiIiYlxNJaRrJe1u+0pJjwHWANheLSmJZkRERGy6MqlplKYS0jcB50taXcc4AkDSA6i679saumdtQ7s2wsyChQYG15WLVbDeJHMK1uycNbtIGM2eWyQOgFfeXSyWStY8LfmFW2o8VslawiVfv1K1QaHg8+rTP/glj8EZA+ViDQ2Wi9ULkpCO0tS57H8o6R+AdbYvkfRwSW8ALrf9b03EjIiIiJgWnM7ikZqaZX888ExgpqTvAfsCFwDHSnq07fc0ETciIiIipp+m2v4PA/YC5gA3ATvZXi7pg8DFQBLSiIiI2DSly36UphLSdbYHgXslXWl7OYDtlZLyLkRERMSmq4/KNXVLU3VI10iaX1/fe3ihpC3o25HmERERETEVTbWQ7md7NYC93vTXWcBLG4oZERER0ftypqZRmpplP2bNIdu3Arc2ETMiIiJiWkiX/SgFC5pNzqrrytTsnC0ViQPAvAXlYs1oajTGaL73zmKxtHDbInG8dk2ROFC25imDher7AqjcMcjArDJxCtYS1swyNXeh8PFeqJZwUQUnqHhdwfeq5Ge44N+s6E09m5BGRERE9CNnlv0oSUgjIiIiSkqX/ShpI4+IiIiIjaqRhFTSVyW9WNKkBk1KWixpqaSlZ95wYxO7FhEREbFxeah7lz7RVAvpY4HnAtdI+pKkQyW1Hclue4ntRbYXvWSHBzW0axEREREb0ZC7d+kTTSWkN9s+DNgV+B/gFcD1kk6X9PSGYkZERETENNRUQmoA28ttn2n7WcBDqc5jf2xDMSMiIiJ639BQ9y59oqlZ9itGLrB9G3BqfWlr3kPK1KrzXXcViQOgFeXqdTJ/YblYpWpAQrHalkVrJZYcA9SH7xVQrIahBgrWBi1Z83SgYMGVUsdFwc+VC8Yq+l6VVPL7ohf0UVd7tzR1pqb9JO1bXfUlkh4OHAhcbvu8JmJGRERExPTUSEIq6XjgmcBMSd+jmuR0PnCspEfbfk8TcSMiIiJ6Xh/Nju+Wptr+DwP2AuYANwE72V4u6YNU40iTkEZERMSmKV32ozQ1aGOd7UHb9wJX2l4OYHslkJ8FEREREXGfplpI10iaXyekew8vlLQFSUgjIiJiE5Zz2Y/WVEK6n+3VAF5/+uEs4KUNxYyIiIjofemyH6WpWfarx1l+K3BrEzEjIiIiYnrq2YJmQ6sGi8TRFlsUiQPAvAXlYpVUsi5eoZmJXnVPkThA0dqgKlmHdHBtuVjrCh0Xs+cViVNayTqarFtTJIwKvlfq1+7XkrVBC9bd7QlpIR2lZxPSiIiIiL6Usk+jbGKnRoiIiIiIXpMW0oiIiIiS0mU/SiMtpJL+puX6LEnHSTpH0omS5k+w3mJJSyUtPeOKG5rYtYiIiIiNykPu2qVfNNVlf0bL9fcCewAnA/OAU8dbyfYS24tsLzpqjx0a2rWIiIiI6CVNddmr5fpTgH1sr5X0I+A3DcWMiIiI6H191LLZLU0lpFtIeh5VYjrH9loA25aUdyEiIiI2Xf1aKmwDNJWQXggcXF//uaTtbS+T9EA6LYxf6NeD71lRJA6A7lleLBabFayvWrJ+XKm6eDMGysTpZyVfw378bi/5B6tgzVjN3axMoILPqWQdVxUsjuNCNWMBVLLmafSkps7UdLSkxwJDti+R9HBJLwIut/2UJmJGRERETAvpsh+lkYRU0vHAM4GZkr4H7AtcABwr6dG239NE3IiIiIiel4R0lKa67A8D9gLmADcBO9leLumDwMVAEtKIiIiIAJpLSNfZHgTulXSl7eUAtldK6sfRXhEREREdsdNCOlJTCekaSfNt3wvsPbxQ0hb05/SDiIiIiM6ky36UphLS/WyvBvD60w9nAS9tKGZERERETENNzbJfPc7yW+m07FNEREREP0oL6ShNtZBuMK8r9GaVrH02Z265WLPmlIs1c3a5WIVqnmr2vCJxoGwNw5K8ZmWxWJq3sFisUjS73PeF1xWsbblmVZE4Kvi9pIFyf0pdsu5zSQVfw17QT+eg75ZUoo2IiIiIjWrT+kkSERERsbGlhXSUtJBGREQSJY3FAAAgAElEQVRElDTUxUsHJB0o6Y+SrpB07Bj3P1jS+ZJ+Jem3kp41xv0rJL1xKk+3E40kpJLmS/o3SW+SNFfSUZLOkfR+SQsmWG+xpKWSlp5x1Q1N7FpERETEJkPSAPAxqjNoPhx4gaSHj3jYccCXbD8aOAL4+Ij7PwR8q8n9bKqF9Axge2A34FxgEfABQMB/jbeS7SW2F9ledNRDdmho1yIiIiI2Hg+5a5cO7AtcYfsq22uAs4DnjNwlYPP6+hbAfa2Ckp4L/B9wWbtAkr4q6SBp8jPGmxpD+le2ny9JwI3AU21b0kXAbxqKGREREdH7ujiGVNJiYHHLoiW2l7Tc3hG4tuX2dcBjR2zmncB3Jb0O2Ax4ar3tBcCbgacBnXTXfxw4GvgPSWcDp9v+YyfPo9ExpK7OjXVe/f/w7YzkjYiIiOiC1t7l+rKk/VqjvAA4w/ZOwLOAM+tWzncCp9he0eG+fN/2i4DHAFcD35f0U0lHS5o10bpNtZAulbTA9grbLxteKGl34O5ONrD29jJ569yZA0XiALC6TP09AFbdUy7WvIJz41Tm/erbWn+Da4uFKlkHkqFCtVxL1owtWCN5Cr1rU1fyuIgNU+pzBXjdmmKxekLZ8tPXAzu33N6pXtbq5cCBALZ/JmkusC1VS+phkt4PbAkMSVpl+z/HCyZpG+DFwEuAXwGfA55AdabOJ4+3XlNnavpHSftKsu1L6sGzBwJ/BJ7YRMyIiIiI6aBwYfxLgD0l7UaViB4BvHDEY64BngKcIelhwFzgFtv35WyS3gmsaJOMfg34a+BM4GDbN9Z3fVHS0ol2spGEVNLxVLO5Zkr6HlWGfT7VOIS9gPc0ETciIiIi7md7naTXAt8BBoDTbF8m6QRgqe1zgH8FPinp9VRDK48aHm45Sf9h+/xx9mPRRCs21WV/GFXiOQe4CdjJ9nJJHwQuJglpREREbKoKnzHa9nnAeSOWvaPl+u+Bv2uzjXd2EOrhkn5l+04ASVsBL7A9sozUKE0NHFpne9D2vcCVtpcD2F5J8bchIiIioncULvtU0iuGk1EA23cAr+hkxaYS0jWS5tfX9x5eKGkLkpBGRERE9KOBuuQncF9R/o5mMjbVZb+f7dUA9nrTUmdRzbKKiIiI2DT1b9Pct6kmMH2ivv3KellbTc2yXz3O8luBW5uIOVV3f+FSFr5wnzLBZhUsd1IwluYtLBaLtYVKg6y8G+ZuVibWQFO/C8dQsrzPwIQl57prRqnnVfD1K6hvy5wVOi5UshpYwbJZJUsxFS091gNKVpAr7M1USeg/1be/B3yqkxUL/iXsTcWS0ZheSiWjEdGMYj9SIqagTxPSulf8v5jgNPHj2eQT0oiIiIjYcJL2BE4CHk5VyxQA2w9pt24S0oiIiIiC+rjL/nTgeOAUYH+q89p31F1RvE9D0lTOsRoRERHRH4a6eOkt82z/AJDtv9S1Sw/qZMVGElJJW49z2QZ41gTrLZa0VNLSz15/43gPi4iIiIjes1rVDLU/S3qtpEOBBZ2s2FSX/S3AXwC1LHN9e7vxVrK9BFgCcPNTntRz1V4jIiIiNlQfd9kfA8wH/hl4N1W3fUflPptKSK8CnmL7mpF3SLq2oZgRERERPa8fE9K6CP4/2H4jsIJq/GjHmkpIPwxsBYxKSIH3d7KB1cvLzLdafeqv2OYFbSd/dYW2L1jrb6jc0e7V9xSLpRmFaluuHbOUbjNK1t+bMVAu1uDacrFc8HmVUrDepArWwnXBv8SlalsWfU4la8YW/DvCrMyxnu5sD0p6wlTXb/tplXSSpM0lzZT0HUnLJL2wzU59DJgjaZ96Gw+X9AZJz7L90anubBNKJaMREVHOplZoPaYXD3Xv0mN+JekcSS+R9LzhSycrdvKT5Jm23yLpucANwBHABcDnx1tB0vHAM4GZkr4HPBY4HzhW0qNtv6eTnYuIiIjoO1b7x0xPc4HbgANalhn4arsVO0lIhx/zLOBs23dIajfh6DBgL2AOcBOwk+3lkj4IXAwkIY2IiIjoI7YnNW60VScJ6bck/Q4YBF4jaVug3QC7dbYHgXslXWl7eb2jK6WSZ/2NiIiI6C092NXeFZJOp2oRXY/tl7Vbt21CavtNkj4A3G57naRVQLvxAGskzbd9L7B3y45uQS+WcY2IiIgoxEN922X/zZbrc4FDqYZ7ttU2IZU0D3gZsAvwT8ADgT2B6ydYbT/bqwG8/vTDWXRYjyoiIiIipg/bX2m9LekLwEWdrNtJl/1pwP8CT6xv3wCcDXxrgh0as0vf9q3ArZ3sWEREREQ/6tcu+zHsyQQnRGrVSUK6p+0XSDocwPa9kvqmrfm2L1zFNs/fpUywtWvKxCkda/a8YqGswSJx1FiJ3jHMKFiepmQpnMGCtVxLmTWnWKiStS1L1t3V3M2KxSpFzZyFe2wDhWoxA/RrzdMe4D6dZS/pbtYfQ3oT8OZO1u3kr+4aSXOHA0jaDSiY7TSrWDIaERER0cdsL5zqup38rDsB+Dawk6TPUNUTfctUA0ZERERsyvq1ML6kQ+sJ7MO3t6zr2LfVySz7b0u6FHg8IOBNtm+e8t5GREREbML6eJb98ba/NnzD9p31yZK+3m7FTge+PAV4pO2vU50SdO+JHixpgaQTJF0m6S5Jt0j6uaSjOowXEREREdPLWHllR5MyOjmX/X8C+wMvrhfdA5zaZrXPAVcBzwDeBfwH8BJgf0knThBrsaSlkpZ+7paJqkpFRERETE929y49ZqmkD0navb58CLi0kxU7aSF9vO1XAqsAbN8OzG6zzq62z7B9ne0PAYfY/jNwNBMU1be9xPYi24te9IAdO9n/iIiIiGnFQ+rapce8jmri+xeBs6hyx9d0smInzahrJc3g/ln229D+bEv3SHqC7YskHQLcDlWR/H4qGRURERERFdv3AMdOZd1OEtKPAV8BHiDpXcDzqbrhJ/JPwCcl7QlcBrwcQNID6u21NTCrzNSxO7/2f2x52B5FYjFQrralXaZeJ1Qz3YrF0kDBaGWoZB3XlXcXi6V5U67+MXmlppqWrJVYcvrsjHKfK5eskdyHVPIYXFeuPm3v9Tw3qwdbNrtC0veAw23fWd/eCjjL9jPardvJLPvP1rPsn0qVexxu+3dt1vmNpJcCOwI/t72iXn6LpD+1fUYFFUtGIyIiIujJsZ/dsu1wMgpg+w5JG36mJlXNUb+1/Qiqls6OSPpn4NXA5cCnJB1j+xv13SdS1TWNiIiIiP4xJOnBtq8BkLQLHTaAT5iQ2h6UdJWkHW1PZtr7K4BFtldI2hX4sqRdbX+Esj28ERERET2lX7vsgbcBF0m6kCrfeyLwyk5W7GRQ4wLgD5J+RlXyCQDb486WB2a0dNNfLenJVEnpLiQhjYiIiE1Yv57Lvj6Z0mOA/1cv+hfbt3aybicJ6b9PYZ+WSdrL9q/rHVwh6dnAacCjprC9iIiIiOhxdQL6TUm7A/8k6Yh66OeExk1IJX3X9tNt/2AK+3MksG7EDq4DjpT0iSlsLyIiIqIv9No56LtF0g7APwAvpGqAPAk4opN1J2ohfcBUd8j2dRPc95OpbjciIiJiuhvqsy57SYuBF1BVV/oSVbnPb9huVyb0PhMlpFtImuisSl/tNMhUDMwp9PNhoGBdy4J1SIua0ckJv7pkVruThHVJwdqgrC1X66/oezW4tlysUnU0S75+awu+frPmlItVSNH6vmtWlotVsmmt4Guofv37uOn4T+BnwAttLwWQNKniVhMmpMCzGXsSkoFGE9KIiIiIftSHk5oeBBwOnCzpgVStpLMms4GJEtK/2H7ZBuxcRERERIzQb2WfbN8GnAqcKmknqnGkyyT9Afia7be228ZE/U/99WpFRERERKNsX2f7ZNuLgOcAqzpZb6KE9CVT3RlJv5R0XD3lPyIiIiJqdvcuvcz2n2yf0Mljx01I252vvo2tgC2B8yX9QtLr61IAE5K0WNJSSUvPvOHGDQgfERER0Zs8pK5d+kVTU0bvsP1G2w8G/hXYE/ilpPPr0gBjsr3E9iLbi16yw4Ma2rWIiIiI6CXjJqSSflD//74NCWD7x7ZfTVWb6n3A4zZkexERERHT2ZDVtUsvUeXFkt5R336wpH07WXeiWfYPkvR44BBJZzFikpPtX06w7p9GLrA9CHy7vrQ1UKjcJIODhQL1sYI1+JhZqF5iyec0Z7NysQryujUbexe6rmitxFLHOsDguvaP6RLNLPPl7nvuLBIHKFcHF/q2nrXXdDTvpW/0YdmnYR8HhoADgBOAu4GvAPu0W3GiI/sdwNuBnYAPjbjPdbAx2T5C0kOpWkUvtr1i+D5JB9ruKCmNiIiIiGnjsbYfI+lXALbvkNTRr9BxE1LbXwa+LOnttt89mb2R9DrgtcAfgE9LOsb2N+q7T6TDVtKIiIiIftPrs+M3wFpJA1QNl0h6AFWLaVtt2/5tv1vSIcB+9aILbH+zzWqLgb1tr5C0K1Viu6vtj5D6phEREbEJ67Wxn130H8DXgO0kvQc4DDiukxXbJqSSTgL2BT5XLzpG0uPbVN2fMdxNb/tqSU+mSkp3IQlpRERERN+x/TlJlwJPocr3nmv7D52s28no6IOAvWwPAUj6DPArYKKEdJmkvWz/ut7BFZKeDZwGPKqTHYuIiIjoR/02qUnS1i03bwa+0Hqf7dvbbaPT6XpbAsMb26KDxx8JrDdt0/Y64EhJn+gwZkRERETf6cMxpJdSjRsdK9M28JB2G+gkIT0J+JWk8+tA+wHHTrSC7esmuO8nHcSMiIiIiGnA9m4buo1OJjV9QdIF3F9D6s22b9rQwO0MFSqL55WrywQCNLNc/bhqklv/8boy71epWokALlkDclafPq9StVwLHX/F9WFtS82eVyxW0WNdTZ1gcbSSz2tT08eTmpD0POAJVC2jP7b99U7W6+hbyPaNwDlT372IiIiIgP4bQzpM0seBPbh/DOmrJD3N9mvardt/P4sjIiIiYmM4AHiY7eE6pJ8BLutkxSSkEREREQX1cZf9FcCDgb/Ut3eul7U14WAUSQOSLp/s3kg6sOX6FpI+Lem3kj4vafvJbi8iIiKiX7iLlx6zEPiDpAvq+Ue/BzaXdI6kCYd+TthCantQ0h8lPdj2NZPYodbTg54M3AgcDDwP+ATw3LFWkrSY6ixPfOCv9+TIHR80iZARERERva+PW0jfMdUVO+my3wq4TNIvgHuGF9o+pMMYi2zvVV8/RdJLx3ug7SXAEoCbn/KkHkz8IyIiImIsti8EkLQ5LTlmtwrjv30K+7SdpDdQ1S3dXJKGB7jSZphARERERD/r41n2i4ETgFXAEFUe2J3C+LYvrM9Bv6ft70uaD7QrcvlJqnEEAJ8BtgVukfRA4NftYgIsv3FuJw/bYAu22apIHADWlavp5jUri8XSwoKvYakafCXr7w3MKhaqX+slFqsPOjRUJg6UrXlasrZlqePdBd+rfq0NWvI1HBosF6sHFHxlS3sT8Ejbt052xbYJqaRXUI3r3BrYHdgROBV4ynjr2H6XpIfWj73Y9op6+U2SPj/ZnYyIiIiInnclcO9UVuyky/41wL7AxQC2/yxpu4lWkPQ64LXAH4BPSzrG9jfqu1snPEVERERsUjzmKd/7wluAn0q6GLivm8f2P7dbsZOEdLXtNVL14kmaSftKA4uBvW2vkLQr8GVJu9r+CPTvuxARERHRzlD/Ttv+BPBD4H+Z5MiEThLSCyW9FZgn6WnAq4H/abPOjJZu+qslPZkqKd2FJKQRERER/WiW7TdMZcVORmIfC9xCle2+EjgPOK7NOsskDZd6ok5On001uelRU9nRiIiIiH4whLp26THfkrRY0oMkbT186WTFTmbZD9XnIr2Yqqv+jy0lnMZzJLDeVEDb64AjJX2ikx2LiIiI6Eelx5DWZ9D8CFWVpE/Zfu+I+08B9q9vzge2s71lfd/7gYOoGjG/BxwzQR74gvr/t7Qs607ZJ0kHUc2qv5Kqu303Sa+0/a3x1rF93QT3/aRdzIiIiIjYcJIGgI8BTwOuAy6RdI7t3w8/xvbrWx7/OuDR9fXHA38H/E1990XAk4ALxople7ep7mcnY0hPBva3fUW9c7sD5wLjJqTdMHfh2iY3fx/fPOlSWVOmhZuXi/XQXYvF4p67ysVaUKjm6aw5ZeIAzChYr7NkHc1ZnXy9dEmp2pYzC75XJY+LkkrV0Rwod/ypYCyvuqf9g7qlYI3kTe2UOYXrkO4LXGH7KgBJZwHPoTrP/FheABxfXzcwF5hN1Sg5C1g2UTBJjwQeXq9XbcT+bLud7ORTdPdwMlq7Cri7g/UiIiIiYoRudtnXZ0da3LJoSX0q9mE7Ate23L4OeOw429oF2I1qpjy2fybpfOBGqoT0P23/YYJ9OR54MlVCeh7wTKpW1aknpJKeV19dKuk84EtUmfLhwCXtNhwRERERzaqTzyVtH9iZI4Av2x4EkLQH8DBgp/r+70l6ou0fj7P+YcDfAr+yfbSk7YH/7iTwRC2kB7dcX0Y1ZgCqGffzOtl4RERERKyvcJf99cDOLbd3qpeN5QiqEyINOxT4+XApT0nfAh4HjJeQrqwnw6+TtDlw84jY4xo3IbV9dCcbGEt9zvrjqV7zdwCvA/6e6sxNx9i+cZz17mt2fu+uf82LttthqrsQERER0ZMKJ6SXAHtK2o0qET0CeOHIB9WnfN8K+FnL4muAV0g6iarL/knAhyeItVTSlsAngUuBFSO2N65OZtnvRpVQ7tr6eNuHTLDaGVQTnzYDzgc+BzwLeC7VjP3njLVSa7PzdY89oH/PYxARERFRgO11kl4LfIeq7NNpti+TdAKw1PY59UOPAM4aUdLpy8ABVLXoDXzb9rgnR7L96vrqqZK+DWxu+7ed7Gcnk5q+Dnya6uxMnSb129v+KICkV9t+X738o5Je3uE2IiIiIvpO6Tqkts+jmmTUuuwdI26/c4z1BqlOijShejLUnbbvqm/vT9UI+RdJl9te024bnSSkq2z/RwePa9VawGHkzKqBSW4rIiIiom8M9dwJljbYl6jGm95Vn6nzbOAkqglOHwf+sd0GOklIP1JP4/8usHp4oe1fTrDONyQtsL3C9n2nGa1na/2xg5jccfNmnTxsg229YH6ROADMKVfb0rfdUCwWs+e2f0yXzChVF2/t6vaPiYmpXGFBr1lZJI4KPqeidUhL1pss9RqWqncKRY91ZqRNJ3rSPNvDiceLqYYFnKzqS/PXnWygk4T0UcBLqMYQDHfZu749JtvvkPRQSTsCFw/PzrJ9haRPdbJjEREREf2oB89Bv6Fan9AB1KcOrWfcd7SBThLSw4GHdNL/f99eVaedei3VrPpPSzrG9jfqu08Evt3ptiIiIiL6SR/O2v6hpC9RFdDfirqwvqQHAR3lj50kpL8DtqSqJdWpxcDetldI2hX4sqRdbX8E+u9nQURERMQm7F+AfwAeBDzB9vD53x8IvK2TDXSSkG4JXC7pEtYfQzpR2acZLd30V0t6MlVSugtJSCMiImITVrgOaePqUlFntS6T9Gzb3+x0G50kpMdPdseAZZL2sv1rgLql9NnAaVRjUiMiIiI2SUMdjquc5k4AupeQ2r5wCjtxJLDeFEfb64AjJX1iCtuLiIiIiOljUll321oVku6WtLy+rJI0KGn5ROvYvs72TePc95PJ7GBEREREP3EXL71A0uH1/7u1LG5bUL9VJy2kC1sCiuq0n/9vMkF62oIF5WLde0+5WAVrg2rewvYP6hKvLvMaav4WReIAMLi2/WO6Zfa8YqG8ruPCHBtMA52MPupGoIL1JovW0RwsF2ug0GtY6pig3PdScQXr02rm7GKxekG/jSGlKvN0NvAV4DEAtn8xmQ1M6hNbD1r9el0o/9jJrBsRERERfek2Sd8FdpN0zsg720yEBzpISCU9r+XmDGARsGoyexkRERERlT48dehBVC2jZwInT2UDnbSQHtxyfR1wNVW3fURERERMUr+dqak+edLPJT3e9i2SFtTLV3S6jU7GkB69Afs4iqRv2X7mOPctpiqqzzu2eRSHLdylm6EjIiIiojnb1133W1NNPboFeKnt37VbcdyEVNI7JljPtt89wbqPGe8uYK8JNroEWALwv7sd3CuTxyIiIiK6po8TnCXAG2yfD1CfGGkJ8Ph2K07UQjrWtMHNgJcD2wDjJqTAJcCFjF2Dast2OxURERHRr/pwDOmwzYaTUQDbF0jarJMVx01Ibd83KFXSQuAY4GiqU0O1G7D6B+CVtv888g5J13ayYxERERExrVwl6e1Uk5sAXgxc1cmKE44hlbQ18AbgRcBngMfYvqOD7b6T8Yvuv66THZs1q0xdvKE/X10kDsDAI/+6WCwK1oC0y9UwVKk6kCVrQA4VrEg3c06xUMVqgwJeW+Z416yCtRJnFKx5OqvccVHsszVY8HNVsF5nv7L7sDLnBPr42b4MeBfw1fr2j+plbU00hvQDwPOo+v4fNZmZUra/LOmhkp4CXDxi3ZSMioiIiE1Wv44hrRst/xlA0oNs39jpuhP9BP9XYAfgOOCGltOH3t3u1KGS/hn4BlVr6O8ktZaJOrHTnYuIiIiIaencyTx4ojGkG9Jf9Apgb9srJO0KfFnSrrY/wtgTnSIiIiI2CX08qanVpJ5lU4O8Zgx309u+up72/2VJu5CENCIiIjZhfTyGtNUnJ/PgpkbNL5N0X73ROjl9NrAt8KiGYkZERETERiJpeHY9tj8+ctlEmmohPZLqNKP3sb0OOFLSJxqKGREREdHz+riF9BGtNyQNAHt3smIjCant6ya47ydNxIyIiIiYDtxngxclvQV4KzCvZeK7gDXUZ+Bsp1yhwEmaO39tkTjaYusicapgBY/AeQuKhVLJGnyFatV5sMzxB6A5HZ3EojsK1vorWVewVH1QF6xPq5kFa56WrLtb6rNVsOZu3ypYC7dYjelohO2TgJMknWT7LVPZRs8mpBERERH9qI+77L8lab+RC23/qN2KSUgjIiIiCurjhPRNLdfnAvsClwIHtFsxCWlEREREbDDbB7felrQz8OFO1u2pQRuSFktaKmnp528bd15URERExLTlLl563HXAwzp5YCMtpJIWAP8G/D2wE9UsqyuBU22fMd56tpdQz8a6eq+nTYPXOSIiImJy+vVMTZI+yv158gxgL+CXnazbVJf954CvAc8Ang9sBpwFHCfpr2y/taG4EREREbFxLG25vg74QqflPptKSHdtaQn9kKRLbL9b0tHA76lqVUVERERscvp4UtMXgT3q61fYXtXpik0lpPdIeoLtiyQdAtwOYHtI6qwY5913zW1o19b3gJWri8QprmBdQa+6p1isYr0cAwWPi5J1XOctLBer5DFYKE7RWokF67jCQLlQQ4We15qVZeIAKvm5Ksjr1pSLtbZP/xaPo98SUkkzgROBlwF/ofpzvbOk04G32W5bgLipb9dXUbWM3kE1lvR19Q4/APhYQzEjIiIiorwPAFsDu9ne2/ZjgN2BLYEPdrKBpk4d+luq2lMjl98i6e4mYkZERERMB304a/vZwF/Zvu+p2V4u6Z+Ay4Fj2m1gY5R9etdGiBkRERHRE4bUvUuPcGsy2rJwkA7z76bKPv12vLuA7ZuIGRERETEd9NsYUuD3ko60/dnWhZJeTNVC2lZTk5q2pyr5dMeI5QJ+2lDMiIiIiCjvNcBXJb2M6lShAIuAecChnWygqYT0m8AC278eeYekCxqKGREREdHz+m0Mqe3rgcdKOgB4RL34PNs/6HQbTU1qevkE972wk23cvWp293ZoAr85bYi/fc28IrG8quNyXBtMBUvuFDV3s429B9030NTvwjEULK2i2WU+VyW5YCmmqopKIUOD5WKVPN4LlVTz6nKl75g5p1yskvrxu30CQ32XklZs/xD44VTW7alz2W8MpZLRiIgoqGR934jYYAV/qkZEREREH05q2mBJSCMiIiIK6s8O+w2zyXfZR0RERMTG1VMJqaTFkpZKWvqNe/9vY+9ORERERNcNdfHSLxpJSCX9UtJxknafzHq2l9heZHvRc+bv1sSuRURERGxUfXimpg3WVAvpVsCWwPmSfiHp9ZJ2aChWRERERExjTU1qusP2G4E3Snoi8ALgl5L+AHzB9pJ2G9h2yzJ13a7/3D3scMj8IrG0cGWROKUVrTc5uLZMnJLPaahgp4vL1actOmh/VqHajAXfK6++t1gszSnzHQhAoVquJWuDFv0OXNOff0dQT40gbFy/1iHdEI0fAbZ/bPvVwI7A+4DHNR1zMkoloxERERFQ/WDv1qVfNNVC+qeRC2wPAt+uLxERERERQEMtpLaPGO8+SUc3ETMiIiJiOsgs+9E2xqCNd22EmBERERE9YQh37dIvGumyl/Tb8e4Ctm8iZkRERERMT02NId0eeAZwx4jlAn7aUMyIiIiIntc/7Zrd01RC+k1gge1fj7xD0gUNxYyIiIjoef009rNbGklIbb98gvte2Mk2/njHVt3boYnifAaedPBtRWLN2HXHInGAwvUSC9b7m1Fm2LPmb1kkDlCsLiMA8xaWi1WyXuLa1UXClKw36XVrisUqeQx6bZnnpbmbFYkDlKuPDDDQVDvSWLFmlYu1rsxnOHpXwSO7N5VKRiMiIiIghfHHssknpBERERElJR0dbdM6V1dERERE9Jy0kEZEREQUlElNozXSQirpwJbrW0j6tKTfSvq8pHHrkEpaLGmppKXfvveKJnYtIiIiYqNyF//1i6a67E9suX4ycCNwMHAJ8InxVrK9xPYi24sOnL9HQ7sWEREREb2kRJf9Itt71ddPkfTSAjEjIiIielK67EdrKiHdTtIbqM7MtLkk2R5uV+6oVXa7GWVqknnVYJE4AJo9p1gsVhWsATl3frlYc8p8jH33rUXiAGjhtsViFa0NWrKGYaHn9f/bu/Nouao67edg5BMAABvKSURBVOPfJzfzQAgQIJBAEMHGRg0QaWkFEQSB9gW1WwXpV1nwmm4VULrpV2x6IWizFqi82ja0rshgKyjSKJBGlEAL6BKFBAwhAzMIgTBPCWS89/f+cfZ1FbduZay9a8jzyaqVqlNV5zmnhlP7nn3Ob0fBep0aUbCOZsG6xaVqdsZrLxfJgcI1T0vWLS65vShUY7pduOxTvVyfgO8B44CxwH8C2wFI2hGoG73JzMzMzLZcuUZqOqfB9Kcl3ZIj08zMzKwTlN4/mk42/zegB7g4Is4bcP83gfelm6OB7SNia0nTgO8AWwG9wLkR8ZMcy9iKsk/nAJe1INfMzMys5Up22UvqAS4CDgOWAHMkzYqIRf2PiYjTah5/CrBPuvk68MmIeFDSTsBdkm6MiKYfE5OlQSppfqO7gIZln8zMzMysqfYHHoqIRwAkXQkcAyxq8PjjgC8DRMQD/RMj4ilJzwITgc5okFI1Oj8AvDRguoDbM2WamZmZtb1mnpomaQYwo2bSzIiYWXN7Z+CJmttLgL9oMK9dgd2AXw1y3/7AcODhzV3mweRqkF4PjI2IuhOYJN2aKdPMzMys7TWzoH1qfM5c7wM3zLHA1RHxhhJEkiYBPwQ+FZnKjeQ6qemkddz3iRyZZmZmZlbnSWBKze3JadpgjgU+VztB0lbAz4EzI+L3WZaQNh7L/oXe4UVyhkwYVSQHIJYtK5alseOKZRWtHzeszOeCnnJfjShUlxFAw8t93kvVmwSgZM3OQqJ3bcGwgrUtC9WnValtBZSt41pw28SwgrWES24v2kDhwvhzgD0k7UbVED0WqNs5KOnPgAnA72qmDQeuAX4QEVfnXMgtqxKtmZmZWYuVHMs+ItYCJwM3AouBqyJioaSvSDq65qHHAlfWDGQE8DHgIOAESfPSZRoZtO0eUjMzMzPbfBFxA3DDgGlnDbh99iDPuxy4POvCJW6QmpmZmRXksezruUFqZmZmVlBfeCz7gbIcQyppR0nfkXSRpG0lnS3pXklXpdIBjZ43Q9JcSXN/sSJLmSszMzMzazO5Tmr6PtUIAE8AtwArgKOA3wDfbfSkiJgZEdMjYvqRo3bPtGhmZmZmrRNNvHSLbCM1RcS/A0j6bEScn6b/u6SGNUrNzMzMul3Jsew7Ra4Gae2e1x8MuK9nQ2Ywmt71P6gJ+patLJID0CMVy2JtwRqGa1YXi4rnG9XybS7tsGuRHABWlKtPS8E6pLHqtWJZxb5ZReu4FvwO95XZ3gJoeJn6oLG63LYdlaugWLSW8NpVxaKK1t21tpSrQXqdpLERsTwi/qV/oqQ3A/dnyjQzMzNre80cOrRb5Bo69KwG0x+S9PMcmWZmZmadwGWf6rVipKZzWpBpZmZmZm0qyx5SSfMb3QXskCPTzMzMrBP4pKZ62c6yBz4AvDRguoDbM2WamZmZtT0fQ1ovV4P0emBsRMwbeIekWzNlmpmZmVkHynVSU8NaoxHxiRyZZmZmZp3AJzXVa9ux7O8ZMaJIzj5FUpKVBWu6vba8WJZGjymXNWm3MkE9w8rkFM6K118plqVhBeslDi2zvWBludqqRWue9pQ7vzVWryiSU7JeZ6wtV4u5pJK1QTW0TH3adhEey75OK86yNzMzMzP7k7bdQ2pmZmbWjXyWfT03SM3MzMwK8jGk9dqqy17SDElzJc29ffmDrV4cMzMzs6aLJv7rFsUbpJJ+0ei+iJgZEdMjYvpfjt2j5GKZmZmZWYvkGqlp30Z3AdNyZJqZmZl1Ah9DWi/XMaRzgNuoGqADbZ0p08zMzKztuexTvVwN0sXA30VE3YGgkp7YkBm8pDJv1qsLyn0oJuz4arGsnkk7Fsvi9XK1GWPFsjJBhWolAkg9xbIYWvA8xpJ1SFcV+gyqYL3OZc8Xy9LYbYplsXplkZiiP/cF65BGT8Hv8JqCtbNL1vi1tpTrk302jY9PPSVTppmZmVnb81n29XINHXr1Ou6ekCPTzMzMrBN009nxzdKKsk/ntCDTzMzMzNpUrrPs5ze6C9ghR6aZmZlZJ/BZ9vVyHUO6A/AB4KUB0wXcninTzMzMrO35LPt6uRqk1wNjI2LewDsk3Zop08zMzMw6UK6Tmk5ax32fyJFpZmZm1gncZV+vYEGzjfOSeovkjJm8tkgOgMYUrMvYW+b1A2BkwfVa9nKRGE3cuUgOACPGlMtaW7KuYLm6uwwdUSZnSLnzQFVqnUobNa5IjArW6yzZtFDJWrhDytVIVqHPRbvwWfb1WnGWvZmZmZnZn7TtHlIzMzOzbtTnk5rqZGuQSnoT8BFgCtALPAD8KCIK9uOZmZmZtRc3R+tl6bKXdCrwXWAk8E5gBFXD9PeSDl7H82ZImitp7oJlD+dYNDMzMzNrM7mOIf00cGRE/CvwfuDPI+JM4Ajgm42eFBEzI2J6REzfe9zumRbNzMzMrHX6iKZdukXOY0iHUnXVjwDGAkTE45KGZcw0MzMza2vd1JBsllwN0ouBOZLuAA4EzgeQNBF4MVOmmZmZmXWgXIXx/03SzcBewAURcV+a/hxw0IbMY2yhilQrni5XZy1+90SxrJEHFItCE7Yul7XNtkVyYsjSIjkAFKwrqK0nFsuK3jXFstSzukzOuO2K5ADEmhXFsli1vFxW9JXJGbNNmRwoWp82Vr1WLIuech2asWJZsax24KFD62Xrso+IhcDCXPM3MzMz60Tusq/nwvhmZmZm1lIujG9mZmZWkIcOrecGqZmZmVlBPoa0nrvszczMzKylvIfUzMzMrCCf1FTPDVIzMzOzgtxlX69tG6STesscTbD0ia2K5ADs+dZy9eM0dky5rEK1QQEYXW69ihkxstVLkIUK1jBk1LgiMRG9RXIA6GnbzfNm0ehC9UEL1veld225rFJ1XClcG7RLP++24fwJMDMzMyvIXfb1svwJKelUSVNyzNvMzMysk0UT/3WLXH0aXwXukPQbSZ9NY9ibmZmZmdXJ1SB9BJhM1TDdD1gk6ZeSPiWp4cFekmZImitp7m+XP5hp0czMzMxapy+iaZdukatBGhHRFxGzI+IkYCfgP4AjqBqrjZ40MyKmR8T0d4/dI9OimZmZmbWOu+zr5TqpSbU3ImINMAuYJWl0pkwzMzMz60C5GqQfb3RHRLyeKdPMzMys7XVTV3uzZGmQRsQDmzuPhT2rm7Eo67XXihFFcgCib3m5rOdeLJbFyIJ1NF8utF7LC9aMnTSpWFZsv3OxLEYW7Ax59fkiMRpdpt5pcQXraPaVeq+22q5ITnGrV5TLcm3QbLqpq71ZPJa9mZmZmbWU//wxMzMzK8hd9vXcIDUzMzMryF329dxlb2ZmZmYt5T2kZmZmZgW5y76eG6RmZmZmBbnLvp677M3MzMyspdp2D+kfC9XsHD9seJEcgBUPl6v1N3bHvmJZPPVMsaghe+5eJmj0mDI5AOPGl8taU6a+b3GlvserVpbJAWLFsmJZRWvGFqptGa+9XCQHQCMLbi+61RZW8zSi4G90h9iyPgFmZmZmLdbnLvs6RRqkkt4D7A8siIjZJTLNzMzMrDNkOYZU0p011z8NXAiMA74s6YwcmWZmZmadICKadukWuU5qGlZzfQZwWEScAxwOHN/oSZJmSJorae7jyx/PtGhmZmZmrdNHNO2yISQdIel+SQ812jEo6WOSFklaKOlHNdN3kTRb0uJ0/9SmvAgD5OqyHyJpAlWDVxHxHEBEvCap4Zk9ETETmAlw1C5HdU+z38zMzKwFJPUAFwGHAUuAOZJmRcSimsfsAXwJeHdEvCRp+5pZ/AA4NyJukjQWyHJGVq4G6XjgLkBASJoUEUvTiihTppmZmVnbK9zVvj/wUEQ8AiDpSuAYYFHNYz4NXBQRL6XlezY99q3A0Ii4KU3PVgIpS4M0IqY2uKsP+HCOTDMzM7NO0MyRmiTNoDo8st/M1OPcb2fgiZrbS4C/GDCbPdO8fgv0AGdHxC/T9Jcl/QzYDbgZOCMiepu2AknRsk8R8Trw6IY8duFrSzIvTeXGsW8rkgNw6EPlxiHYa9dyNfh6dhhbLGvtnfcUyenZZcciOQCx5KliWdp6q3JZ4wvWV+0rU9MvRowskgNAT0+5rBVl6j4D5Wqeqtz2NpaV294WrRm7tmDd4pI1kt9xVLmsAmoPd9wMQ4E9gIOBycCvJb0tTT8Q2Ad4HPgJcAJwyWbm1fFITWZmZmYFRRP/bYAngSk1tyenabWWALMiYk1EPAo8QNVAXQLMi4hHImItcC2w72a/AINwg9TMzMysoMJln+YAe0jaTdJw4Fhg1oDHXEu1dxRJ21F11T+Snru1pInpcYfwxmNPm8YjNZmZmZkVVHKkpohYK+lk4Eaq40MvjYiFkr4CzI2IWem+wyUtAnqBf4qIFwAknQ78jyRRnbD+vRzL6QapmZmZWReLiBuAGwZMO6vmegD/kC4Dn3sT8Pbcy+gGqZmZmVlB3TTCUrO4QWpmZmZWUDPLPnWLtm2QThxepmTMNasf54vsUiTrhb4RRXIA1r7ccECsphsyoenlyBqKVWuK5PQ980KRHACNH1Mua5tti2XFkwNP4sxHU6eWCSpZcmf1ynJZY8qVA+P1giWmSpXpGl2u9F3R8kjDC5Y5G1Hwu2VtqW0bpKWUaoyamVlBJWvGmm0kd9nX2+IbpGZmZmYllTzLvlNka5BK2p/qxK05aSzUI4D70pleZmZmZmZApgappC8DRwJDJd1ENWbqLcAZkvaJiHNz5JqZmZm1O3fZ18u1h/RvgGnACOBpYHJEvCrpG8AdwKANUkkzgBkAu2z1ZiaOLjeeuJmZmVkJPsu+Xq6hQ9dGRG9EvA48HBGvAkTECqCv0ZMiYmZETI+I6W6MmpmZmW0Zcu0hXS1pdGqQ7tc/UdJ41tEgNTMzM+t24ZOa6ijHcQySRkTEqkGmbwdMioh71zePocN3LvZuqVDOhdu/r1ASvNhTLIr9VparebrL+FeL5PT1lvpUwMQ3lavL2FOw1N+Q0bk6YOppaJksjS5YmKSv3A+WRg3vuiwNLbgRLGn0qHJZveVqTGtEuc/g6C9eVm4D38CoUbs27Qu+YsUfW74+zZBl6zpYYzRNfx54PkfmpuqKd9HMzMysg7kOqZmZmVlBPsu+nhukZmZmZgX5GNJ65Q7yMjMzMzMbhPeQmpmZmRXkLvt6bpCamZmZFeQGaT132ZuZmZlZS3kPqZmZmVlB3j86iIjoqgswo5tynNVZWd24Tt2a1Y3r5KzOyXFW5+T4UubSjV32M7osx1mdldWN69StWd24Ts7qnBxndU6OFdCNDVIzMzMz6yBukJqZmZlZS3Vjg3Rml+U4q7OyunGdujWrG9fJWZ2T46zOybEClA4MNjMzMzNriW7cQ2pmZmZmHcQNUjMzMzNrqa5pkEq6VNKzkhZkzpki6RZJiyQtlPT5jFkjJd0p6Z6UdU6urJTXI+kPkq7PnPOYpHslzZM0N3PW1pKulnSfpMWSDsiU85a0Pv2XVyV9IUdWyjstfSYWSPqxpJGZcj6fMhY2e30G+85K2kbSTZIeTP9PyJj10bRefZKmNyNnHVlfT5/B+ZKukbR1xqyvppx5kmZL2ilXVs19/ygpJG2XI0fS2ZKerPl+HbW5OY2y0vRT0vu1UNLXcmVJ+knNOj0maV7GrGmSft+/3ZW0f6acd0j6XdrG/7ekrTY3J8130N/eXNsMa4FWF0Jt1gU4CNgXWJA5ZxKwb7o+DngAeGumLAFj0/VhwB3AuzKu2z8APwKuz/waPgZsV+hz8Z/A/0nXhwNbF8jsAZ4Gds00/52BR4FR6fZVwAkZcvYGFgCjqUZ1uxl4cxPnX/edBb4GnJGunwGcnzFrL+AtwK3A9MzrdTgwNF0/P/N6bVVz/VTgu7my0vQpwI3AH5vxvW6wTmcDpzfrPVpP1vvSZ31Eur19ztev5v4LgLMyrtds4Mh0/Sjg1kw5c4D3pusnAl9t0joN+tuba5vhS/lL1+whjYhfAy8WyFkaEXen68uAxVQNhBxZERHL081h6ZLlLDRJk4G/Ai7OMf9WkDSeaoN5CUBErI6IlwtEHwo8HBF/zJgxFBglaShVg/GpDBl7AXdExOsRsRa4DfhIs2be4Dt7DNUfEaT/P5QrKyIWR8T9zZj/BmTNTq8hwO+ByRmzXq25OYYmbTPWsY39JvB/C+Q0XYOszwDnRcSq9JhnM2YBIEnAx4AfZ8wKoH9v5XiasM1okLMn8Ot0/Sbgrzc3J2U1+u3Nss2w8rqmQdoKkqYC+1DtucyV0ZO6cZ4FboqIXFnfovpR6cs0/1oBzJZ0l6ScI23sBjwHXJYORbhY0piMef2OpUk/LIOJiCeBbwCPA0uBVyJidoaoBcCBkraVNJpqr8qUDDm1doiIpen608AOmfNa4UTgFzkDJJ0r6QngeOCsjDnHAE9GxD25MmqcnA5FuDRzt+yeVJ/7OyTdJumdGbP6HQg8ExEPZsz4AvD19Ln4BvClTDkLqRqJAB8lwzZjwG/vlrDN2CK4QbqJJI0Ffgp8YcAeiaaKiN6ImEa1R2V/SXs3O0PSB4FnI+KuZs+7gfdExL7AkcDnJB2UKWcoVXfSdyJiH+A1qi6dbCQNB44G/itjxgSqDf5uwE7AGEl/2+yciFhM1b08G/glMA/obXbOOvKDTD0CrSLpTGAtcEXOnIg4MyKmpJyTc2SkP1L+mYwN3hrfAXYHplH9EXZBxqyhwDbAu4B/Aq5KezBzOo6Mf8QmnwFOS5+L00g9RxmcCHxW0l1UXeurmznzdf32duM2Y0viBukmkDSM6gtxRUT8rERm6mq+BTgiw+zfDRwt6THgSuAQSZdnyAH+tIevvyvsGmCzD65vYAmwpGav8tVUDdScjgTujohnMma8H3g0Ip6LiDXAz4C/zBEUEZdExH4RcRDwEtVxWzk9I2kSQPq/Kd2l7UDSCcAHgePTD2cJV9CkLtNB7E71R9E9adsxGbhb0o7NDoqIZ9If533A98i3zYBqu/GzdMjUnVS9Rpt9slYj6bCbjwA/yZWRfIpqWwHVH8xZXsOIuC8iDo+I/aga2Q83a94Nfnu7dpuxpXGDdCOlv5QvARZHxP/LnDWx/2xcSaOAw4D7mp0TEV+KiMkRMZWqu/lXEdH0PW4AksZIGtd/nepkjyyVESLiaeAJSW9Jkw4FFuXIqlFiT8fjwLskjU6fx0OpjqdqOknbp/93ofrR/FGOnBqzqH44Sf9flzmvCElHUB0Sc3REvJ45a4+am8eQYZsBEBH3RsT2ETE1bTuWUJ108nSzs/obHMmHybTNSK6lOrEJSXtSnQz5fMa89wP3RcSSjBlQHTP63nT9ECDL4QE124whwL8A323SfBv99nblNmOL1Oqzqpp1oWoELAXWUG0YT8qU8x6qLoH5VF2Y84CjMmW9HfhDylpAk87AXE/mwWQ8yx54E3BPuiwEzsy8PtOAuek1vBaYkDFrDPACML7A+3QOVUNjAfBD0hnBGXJ+Q9WIvwc4tMnzrvvOAtsC/0P1Y3kzsE3GrA+n66uAZ4AbM2Y9BDxRs81o1pnvg2X9NH0u5gP/DeycK2vA/Y/RnLPsB1unHwL3pnWaBUzK+PoNBy5Pr+HdwCE5Xz/g+8DfNyNjPev1HuCu9F2+A9gvU87nqXpSHgDOI40I2YSsQX97c20zfCl/8dChZmZmZtZS7rI3MzMzs5Zyg9TMzMzMWsoNUjMzMzNrKTdIzczMzKyl3CA1MzMzs5Zyg9Ssw0gKSRfU3D5d0tlNmvf3Jf1NM+a1npyPSlos6ZYB06dKWiFpXs1l+CbMf6qkTzRviTco8wRJFw6Ydquk6en6Pw+47/b0/1RJC9L16ZK+3YRlKfI+mpk1ixukZp1nFfARSdlGj9kUacSZDXUS8OmIeN8g9z0cEdNqLpsy9OBUYKMbpJJ6NiFrQ72hQRoRdaNrRcTciDg14zKYmbUlN0jNOs9aYCbVeNRvMHDPmKTl6f+DJd0m6TpJj0g6T9Lxku6UdK+k3Wtm835JcyU9IOmD6fk9kr4uaY6k+ZL+rma+v5E0i0FGwZJ0XJr/Aknnp2lnURW5vkTS1zdkhdMIX5em5f2DpGPS9Kkp/+506W/knQccmPawnjZw76Wk6yUd3P8aSbpA0j3AAZL2S6/VXZJurBmW8FRJi9L6X7khy12Tdx4wKi3PFf25gzzuYEnXp+s31OwlfkXSp9bxPkjShZLul3QzsP3GLJ+ZWattzB4NM2sfFwHzJX1tI57zDmAv4EXgEeDiiNhf0ueBU4AvpMdNpRrnenfgFklvBj4JvBIR75Q0AvitpNnp8fsCe0fEo7VhknYCzgf2A14CZkv6UER8RdIhwOkRMXeQ5dxd0rx0/bcR8TngTKohbU9UNZzunanh9SxwWESsVDVk5o+B6cAZaf79DeoT1vG6jAHuiIh/VDVW9m3AMRHxnKSPA+cCJ6Z57hYRq9IybLCIOEPSyRExbSOec1Ra9v2Ay6hGGjuJwd+HfYC3AG8FdqD64+DSjVlGM7NWcoPUrANFxKuSfgCcCqzYwKfNiYilAJIeBvoblPeSxu5OroqIPuBBSY8AfwYcDry9Zu/reGAPYDVw58DGaPJO4NaIeC5lXgEcRNWwWpeHB2m4HQ4cLen0dHsksAvV+NwXSpoG9AJ7rmfeg+mlGm4Tqkbd3sBNkgB6qIZGhGrIwiskXdtgHRoNe7fJw+GlwzJ+CHwsIl6R1Oh9OAj4cUT0Ak9J+tWmZpqZtYIbpGad61tUY21fVjNtLelQHElDqMbl7req5npfze0+3rgtGNiACkDAKRFxY+0dqdv7tU1b/I0i4K8j4v4B+WdTjUP/Dqr1Xtng+X96XZKRNddXpoZcf87CiDhgkHn8FVXD738BZ0p6W0Ssrbn/BWDCgOdsAzzfaKXWJR3PeiXwlYhYULN8g70PR21KhplZu/AxpGYdKiJeBK6i6sbt9xhVFznA0cCwTZj1RyUNSceVvgm4H7gR+Ezq0kbSnpLGrGc+dwLvlbRdalwdR9UdviluBE5R2m0paZ80fTywNO3R/d9UezQBlgHjap7/GDAtrdcUqkMSBnM/MFHSASlnmKQ/T437KRFxC/DFlDt2wHPnAO+WtGN67nRgBPBEun9N/+u3gc4D5kdE7fGqjd6HXwMfT8eYTuKNe7zNzNqe95CadbYLgJNrbn8PuC6doPNLNm3v5eNUjcmtgL9Px2deTHVs6d2pUfgc8KF1zSQilko6A7iFas/ezyPiuk1YHoCvUu0Rnp8ah48CHwT+A/ippE/yxvWdD/Sm1+H76bmPUh1buZhqz/Jgy7w6dYd/W9J4qm3kt4AHgMvTNAHfjoiXBzz3mXQ87g1pGZcDx6XGMlQnos2XdHdEHL8B63w6sLDmeNqzgEbvwzXAIWn9Hgd+twHzNzNrG4rY5MObzMzMzMw2m7vszczMzKyl3CA1MzMzs5Zyg9TMzMzMWsoNUjMzMzNrKTdIzczMzKyl3CA1MzMzs5Zyg9TMzMzMWur/A3mM1cjdkHJvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x504 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(12,7)) \n",
    "sns.heatmap(pivot_results.iloc[50:, 0:20], ax=ax, cbar_kws={'label': 'Out-of-Sample Accuracy'})\n",
    "plt.title('Relationship: Number of Trees, to Number of Features Utilized.')\n",
    "plt.ylabel('Number of Trees')\n",
    "plt.xlabel('Number of Features Utilized')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(b) Could the number of trees be too small for the number of features used?**\n",
    "\n",
    "Yes, having many diversified models allows the ensemble to reduce the model variance and increase the accuracy. When there are too few trees then this effect can't take place. This is especially true when we limit the number of features per tree. \n",
    "\n",
    "**(c) Could the number of trees be too high for the number of observations available?**\n",
    "\n",
    "Yes, they could be too high. Quoting the [post](https://stats.stackexchange.com/questions/36165/does-the-optimal-number-of-trees-in-a-random-forest-depend-on-the-number-of-pred/36183): \"*If the number of observations is large, but the number of trees is too small, then some observations will be predicted only once or even not at all.*\". If the number of trees is much larger than the amount of data available then it will start to reduce the number diversified trees. Some trees will be trained on the same data.\n",
    "\n",
    "\n",
    "## Question 5\n",
    "**How is out-of-bag accuracy different from stratified k-fold (with shuffling) cross-validation accuracy?**\n",
    "\n",
    "Out-of-bag accuracy is the mean accuracy of the models trained on data that didn’t form part of the bootstrap sample. Stratified k-fold cross-validation accuracy is the mean accuracy of the models, scored on an out-of-sample data set. The stratified component is a technique to create k-folds, where each fold is representative of the dataset. It is typically used on data that has a class imbalance.\n",
    "\n",
    "With respect to financial machine learning, because the data isn’t IID and it has a low average uniqueness, a high observation redundancy, bagging models will suffer from trees that are very similar in nature, thus causing the ensemble to be less effective. This problem extends to the out-of-bag accuracy when we sample with replacement as the out-of-bag sample will contain data that is very similar to the observations contained in the bootstrapped sample. It is for this reason that the out-of-bag accuracy is inflated.\n",
    "\n",
    "The solution de Prado suggests is to use stratified k-fold cross-validation without shuffling, before partitioning. If we were to apply shuffling then the redundant observations would spill over to the various k-folds and our model accuracy would be over-inflated.\n",
    "\n",
    "\n",
    "\n",
    "---\n",
    "The following shows how to build ensembles models which take into account average uniqueness, its implementation can be found in the mlfinlab package.\n",
    "\n",
    "```python\n",
    "# Snippet 6.2 Three Ways of Setting up a RF\n",
    "random_forrest = RandomForestClassifier(n_estimators=1000, class_weight='balanced_subsample', criterion='entropy')\n",
    "\n",
    "base_trees = DecisionTreeClassifier(criterion='entropy', max_features='auto', class_weight='balanced')\n",
    "bagging_ensemble = BaggingClassifier(base_estimator=base_trees, n_estimators=1000, max_samples=avgU)\n",
    "\n",
    "rf_1_tree = RandomForestClassifier(n_estimators=1, criterion='entropy', bootstrap=False, class_weight='balanced_subsample')\n",
    "bagging_ensemble_2 = BaggingClassifier(base_estimator=rf_1_tree, n_estimators=1000, max_samples=avgU, max_features=1.0)\n",
    "```\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
